Lectures on Probability, Entropy, and Statistical Physics

Preface Science consists in using information about the world for the purpose of predicting , explaining, understanding, and/or controlling phenomena of interest. The basic difficulty is that the available information is usually insufficient to attain any of those goals with certainty. In these lectures we will be concerned with the problem of inductive inference , that is, the problem of reasoning under conditions of incomplete information. Is there a general method for handling uncertainty? Or, at least, are there rules that could in principle be followed by an ideally rational agent when discussing scientific matters? What makes one statement more plausible than another? How much more plausible? And then, when new information is acquired how does it change its mind? Or, to put it differently, are there rules for learning? Are there rules for processing information that are objective and consistent? Are they unique? And, come to think of it, what, after all, is information? It is clear that data " contains " or " conveys " information, but what does this precisely mean? Can information be conveyed in other ways? Is information some sort of physical fluid that can be contained or transported? Is information physical ? Can we measure amounts of information? Do we need to? Our goal is to develop the main tools for inductive inference – probability and entropy – and to illustrate their use in physics. To be specific we will concentrate on examples borrowed from the foundations of classical statistical physics, but this is not meant to reflect a limitation of these inductive methods, which, as far as we can tell at present are of universal applicability. It is just that statistical mechanics is rather special in that it provides us with the first examples of fundamental laws of physics that can be derived as examples of inductive inference. Perhaps all laws of physics can be derived in this way. The level of these lectures is somewhat uneven. Some topics are fairly advanced – the subject of recent research – while some other topics are very elementary. I can give two related reasons for including the latter. First, the standard education of physicists includes a very limited study of probability and even of entropy – maybe just a little about errors in a laboratory course, or maybe a couple of lectures as a brief mathematical prelude to statistical mechanics. The result is a widespread …